Minimization of Information Loss through Neural Network Learning

In this article, we explore the concept of minimization of information loss (MIL) as a a target for neural network learning. We relate MIL to supervised and unsupervised learning procedures such as the Bayesian maximum a-posteriori (MAP) discriminator, minimization of distortion measures such as mean squared error (MSE) and cross-entropy (CE), and principal component analysis (PCA). To deal with unsupervised systems where complex noise is present, we introduce the idea of the signal being well-mixed with the noise. If this holds, minimizing information loss about the pair ((; X i) will proportionately minimise information loss about itself. This situation may hold in early processing stages of complex sensory systems such as the retina in higher mammals.